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Foreword 


In  December  of  1 997,  the  Army  Research  Institute  launched  a  review  of  our 
survey  programs.  ARI  has  been  conducting  surveys  of  soldiers  for  more  than  50 
years,  since  the  Army  survey  program  was  established  by  Gen.  George  C.  Marshall  in 
1943. 


This  review  had  three  purposes:  to  examine  ARI’s  wide  range  of  surveys  and 
methodologies;  to  determine  whether  ARI  could  benefit  by  adopting  some  of  the 
methodologies  used  by  other  survey  organizations  and  new  emerging  technologies; 
and  to  make  recommendations  to  ensure  ARI’s  surveys  and  methodologies  continue  at 
the  leading  edge  and  remain  in  line  with  the  best  practices  of  other  organizations. 

We  selected  The  Gallup  Organization  to  carry  out  this  independent  audit. 
Gallup  used  its  internal  experts,  as  well  as  a  panel  of  internationally  renowned  outside 
experts  in  survey  methods,  to  assess  ARI’s  survey  programs.  We  were  impressed 
with  the  quality  of  Gallup’s  effort  did  and  I  am  pleased  to  share  with  you  their 
findings. 


Dr.  Edgar  M.  Johnson 
Director,  Army  Research  Institute 
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Gallup’s  Report 
Card 

ARl  is  doing  a  good  job. 
Its  surveys  are  done 
efficiently  and  yield  useful 
results. 


Executive  Summary 

Gallup  gives  ARI  high  marks,  finding  that  ARI  uses  sound  methods  to 
design  and  carry  out  surveys  that  are  comparable  to  the  methods  used  by 
other  survey  organizations.  In  addition,  ARI  surveys  achieve  response 
rates  similar  to  those  of  comparable  surveys  in  the  field.  Gallup  reports 
that  ARI’s  customers  express  a  high  level  of  satisfaction  with  ARI 
surveys.  Although  we  found  that  ARI  is  doing  a  good  job,  we  made 
several  recommendations  for  incremental  improvements. 


Gallup  presented  a  number  of  recommendations  to  ARI  which  are 
explained  in  further  detail  in  this  report.  These  include  suggestions  for 
j  improving  response  rates,  standardizing  pretesting  procedures,  adopting 
I  new  data  collection  technologies,  creating  a  quick  turnaround  capability, 
i  standardizing  deliverables,  and  obtaining  systematic  customer  feedback. 

I  In  many  cases,  these  recommendations  support  initiatives  that  ARI  has 
already  begun. 
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The  Task 


Assignment 


Gallup,  one  of  the  oldest  & 
best-known  names  in 
survey  research,  carried 
out  an  independent  review 
of  ARI’s  survey  programs. 
Gallup  also  reviewed  the 
best  practices  of  other 
survey  organizations  as  a 
basis  for  comparison. 


The  purpose  of  Gallup’s  independent  review  of  ARI  was  to  assess  the 
quality  of  ARI’s  current  survey  programs  and  to  make  recommendations 
for  improvement.  We  were  also  asked  to  revise  the  Army’s  regulations 
guiding  ARI’s  survey  programs,  incorporating  our  recommendations  and 
bringing  the  regulations  up  to  date. 

In  the  course  of  our  review,  we  gathered  information  about  several  types 
of  ARI  surveys,  including  opinion,  command  climate,  and  occupational 
analysis  studies.  We  studied  survey  documentation  and  interviewed  the 
staff  members  who  carry  out  the  studies.  Gallup  also  gathered  similar 
information  about  a  number  of  comparable  surveys  done  by  the  other 
services  (Department  of  Defense,  Navy,  Air  Force),  academic  survey 
organizations  (National  Opinion  Research  Center,  Institute  for  Social 
Research  at  the  University  of  Michigan),  and  private  firms  (Gallup, 
Personnel  Decisions  Research  Institute,  and  Harris).  In  addition,  after 
identifying  users  of  ARI  data,  Gallup  conducted  both  a  focus  group  and 
telephone  interviews  to  assess  customer  satisfaction  with  ARI’s  surveys. 


Once  the  data  were  gathered,  Gallup 
conducted  a  briefing  for  the  Study 
Advisory  Group,  headed  by  Lt.  Gen. 
Ohle,  the  Army’s  Deputy  Chief  of 
Staff  for  Personnel.  We  also 
completed  a  detailed  report  of  our 
findings,  entitled  “Modernizing  the 
U.S.  Army  Research  Institute’s 
Attitude  and  Opinion  Survey 
Programs.” 
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ARI  Surveys 

Gallup  examined  several 
ARI  surveys  encompassing 
attitude  and  opinion 
surveys,  occupational 
analysis  studies,  and 
special  attitude  surveys. 


ARI’s  Major  Survey  Products 

ARI  conducts  a  wide  range  of  surveys  for  its  clients  in  the  Army  and  the 
Department  of  Defense.  These  studies  fall  into  three  main  categories: 

1)  continuing  attitude  surveys,  which  are  done  on  a  regular  basis  to  track 
the  attitudes  and  opinions  of  soldiers;  2)  special  attitude  surveys,  which 
are  one-time  surveys  carried  out  as  new  issues  arise;  and  3)  occupational 
analysis  studies,  which  gather  information  about  the  specific  tasks 
associated  with  particular  job  categories.  The  special  attitude  surveys 
have  covered  topics  ranging  from  new  recruit  values  to  attitudes  of  Army 
families  to  retention  in  the  Military  Police  Corps. 


ARI’s  main  services  include  clarification  of  the  issues,  development  of 
the  questionnaire,  sampling,  collection  and  processing  of  the  data,  and 
reporting  survey  results. 


Sample  Survey  Command  Survey 

of  Military  Climate  of  Officer 

Personnel  Survey  Careers  j 


Army  Personnel 
Survey  Program 


Occupational 

Analysis 


Special  Surveys 


Survey  of  Survey  of  Officer 

New  Recruit  Army  Evaluation 

Values  Families  |  Report  Survey 
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Covering  the 
Army 

A  survey  can  only  be 
accurate  if  it  covers  every 
portion  of  the  population 
of  interest,  whether  that  is 
officers,  enlisted 
personnel,  spouses  of 
soldiers,  nurses,  or 
chaplains.  Errors  are 
introduced  into  the  survey 
results  when  part  of  the 
population  is  completely  | 
left  out,  i 

! 

1 

i 


What  ARI  Does: 

Many  ARI  surveys  rely  on  the  Army  officer  and  enlisted  master  files  for 
selecting  the  sample.  These  files  also  provide  the  addresses  of  soldiers 
selected  for  the  survey.  These  files  are  good  starting  points  for  ARI’s 
surveys  because  they  include  information  on  all  soldiers  and  officers. 
Because  ARI  mails  some  of  its  surveys  directly  to  the  members  of  the 
sample,  it  is  important  that  the  addresses  be  up-to-date.  Unfortunately, 
this  is  not  always  the  case.  Soldiers  are  a  young  and  mobile  population 
and  it  can  be  hard  to  keep  track  of  their  current  addresses.  This  means 
that  when  the  questionnaires  are  mailed  out,  many  of  those  selected  to 
participate  in  the  survey  never  receive  them. 

What  Other  Organizations  Do: 

Gallup  found  such  problems  to  be  very  common.  A  major  reason  is  that, 
in  both  organizations  and  households,  people  tend  to  be  mobile.  This  is 
especially  true  in  the  Department  of  Defense.  The  best  way  to  reduce  the 
resulting  problems  in  locating  members  of  the  sample  is  to  keep  up-to- 
date  records  of  the  location  of  personnel. 

Gallup  Recommendations: 

It  is  possible  that  the  inaccurate  address  information  in  the  Army’s 
personnel  management  master  files  has  little  effect  on  the  survey  results, 
or  it  may  have  a  large  effect.  Gallup  recommends  that  ARI  undertake  a 
special  study  to  determine  how  extensive  the  current  address  problem  is, 
and  how  undeliverable  mail  affects  the  survey  results.  This  study  would 
help  determine  whether  it  is  worth  the  cost  to  the  Army  to  try  to  improve 
the  list  or  to  move  to  a  different  method  of  data  collection. 
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Questionnaire 

Design 

The  results  of  a  survey  are 
only  as  accurate  as  the 
answers  to  specific 
questions.  The  wording 
and  formatting  of  the 
questions  can  affect  the 
quality  of  the  data 
collected.  As  a  result, 
survey  researchers  have 
developed  a  range  of 
techniques  for  developing 
and  testing  survey 
questions. 


What  ARI  Does: 

ARI  surveys  involve  distributing  a  questionnaire  to  members  of  the 
sample  who  answer  the  questions  and  send  the  survey  forms  back  to 
ARI.  This  method  puts  a  premium  on  designing  easy-to-understand, 
easy-to-follow  questionnaires.  ARI  customers  appear  to  be  quite 
satisfied  with  ARI’s  survey  development  procedures.  More  than  3  out  of 
5  customers  interviewed  said  they  were  “very  satisfied”  with  ARI  on  the 
development  of  the  survey  questionnaire  (62%).  While  pretesting  survey 
questions  is  a  common  practice  at  ARI,  it  is  not  a  standard  procedure  for 
all  of  ARI’s  surveys.  In  addition,  items  that  have  been  used  for  many 
years  need  to  be  re-evaluated  periodically  to  make  sure  they  are  still  up- 
to-date. 


What  Other  Organizations  Do: 

Pretesting  is  a  common  practice  in  the  survey  organizations  Gallup 
examined.  Occasionally,  survey  organizations  also  undertake  special 
studies  to  assess  the  accuracy  of  survey  responses.  Over  the  past  1 5 
years,  there  has  been  a  revolution  that  has  introduced  new  methods 
for  testing  survey  questions,  including  the  use  of  focus  groups, 
cognitive  interviews,  and  small  pilot  studies. 

Gallup  Recommendations: 

Gallup  recommends  that  ARI  strengthen  and  standardize  its 
pretesting  procedures.  More  specifically,  Gallup  recommends  that 
all  new  items  be  pretested  with  nine  or  ten  respondents.  In  addition, 
automated  questionnaires  should  undergo  small-scale  “usability”  tests 
to  make  sure  that  respondents  can  easily  complete  the  survey.  Finally, 
items  that  have  been  used  in  the  past  should  be  subjected  to  periodic 
reevaluation.  Expert  review  panels  are  a  low-cost  method  for 
improving  questionnaires  and  might  be  adopted  as  part  of  the  standard 
proeedure  for  developing  new  items  or  new  questionnaires. 
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Response  Rates  What  ARI  Does: 

.  , .  .  , .  ,  !  ARI  typically  distributes  anonymous  questionnaires.  Because  it  cannot 

Achieving  a  high  response  \  -i 

rate  is  the  only  way  to  {  identify  who  has  returned  a  questionnaire,  ARI  cannot  target  fbllow-up 
guarantee  that  the  data  \ 

reflect  the  population  that  I  efforts  specifically  to  those  who  have  not  yet  completed  the  survey. 

was  sampled.  Data  ;  Given  this  method  of  contact  and  the  minimal  follow-up  of  a  reminder 

collection  methods  and 

operations  have  a  direct  postcard,  ARJ’s  response  rates  are  quite  respectable. 

impact  on  survey  response 
rates.  j 

What  Other  Organizations  Do: 

Response  rates  are  declining  throughout  the  survey  research  industry. 


Survey  organizations  have  tried  to  counteract  this  by  improving  their 


efforts  at  persuasion  (via  such 
methods  as  offering  incentives) 
and  by  making  more  attempts  to 
contact  the  sample  members. 
Follow-up  efforts,  such  as 
reminder  postcards  and  second 
mailings  of  questionnaires  are 
widely  used  to  combat 
nonresponse  in  mail  surveys. 

Gallup  Recommendations: 

Gallup  encourages  ARI  to 
experiment  to  find  the  best 
combination  of  changes  to 
maintain  and  improve  response 


rates.  These  might  include  sending  advance  letters;  developing 
questionnaires  that  are  easier  to  understand  and  complete;  improving 
the  cover  letter  included  with  the  questionnaire;  or  sending  replacement 
questionnaires  or  reminder  postcards  to  specific  individuals.  It  is 
possible  to  carry  out  some  forms  of  follow-up  without  sacrificing  the 
anonymity  of  the  respondents.  Such  follow-up  can  help  improve 


response  rates. 
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Collecting  the 
Data 

The  method  of  collecting 
the  data  can  have  a  large 
effect  on  people ’s 
willingness  to  participate 
and  on  the  answers  they 
provide. 


What  ARI  Does: 

ARl  surveys  are  mostly  self-administered.  Paper  questionnaires  are 
either  distributed  through  local  Army  distribution  channels  or  are  mailed 
directly  to  respondents.  The  data  are  then  scanned  in  by  machine.  ARI 
has  begun  to  incorporate  new  data  collection  technologies,  such  as  disk- 
by-mail  and  other  forms  of  computer  administration,  in  its  data  collection 
efforts. 


What  Other  Organizations  Do: 

Most  of  the  comparison  surveys  Gallup  examined  use  some  form  of 
computer  assisted  interviewing,  ensuring  that  the  right  questions  are 
administered  in  the  right  order  and  that  question  options  are  rotated 
appropriately.  Although  most  surveys  outside  of  ARI  are  currently 
conducted  by  telephone,  many  organizations  have  begun  to  adopt  new 
methods  of  data  collection,  based  on  emerging  technologies  such  as 
touch- tone  data  entry,  voice  recognition,  e-mail  and  Web.  Some  surveys 
use  a  combination  of  methods  to  collect  data. 


Gallup  Recommendations: 

Gallup  recommends  ARI  continue  to 
accelerate  its  adoption  of  the  latest 
data  collection  tools.  Several  of 
these  methods  may  be  particularly 
well-suited  for  collecting  data  from 
subpopulations  of  soldiers  with  direct 
access  to  computers  or  telephones. 
The  transition  to  these  new  modes  of 
data  collection  may  require  a 


substantial  testing  effort  to  determine  whether  the  mode 
itself  alters  the  answers  to  the  questions. 


PAGE? 


THE  GALLUP  ORGANIZATION 


Processing  the 
Data 

Data  processing  is  the 
creation  of  a  dataset  from 
raw  survey  data.  The  data 
is  edited,  cleaned,  and 
weighted  to  remove 
erroneous,  highly  suspect, 
or  missing  values. 


What  ARI  Does: 

The  “bubble-sheet”  questionnaires  used  for  most  ARI  surveys  are 
scanned  to  create  a  raw  data  file.  That  raw  data  is  then  loaded  into 
statistical  software  for  cleaning,  editing,  weighting  and  coding  of  open- 
ended  items.  For  some  surveys,  outside  contractors  process  the  data. 

What  Other  Organizations  Do: 

Computerization  in  data  processing  reduces  the  time  and  expense  of 
coding  the  data  and  compiling  a  usable  data  set.  For  self-administered 
surveys,  modem  scanning  equipment  has  advanced  beyond  traditional 
“bubble-sheet”  scanners  to  Optical  Character  Recognition  (OCR)  and 
Intelligent  Character  Recognition  (ICR)  scanners  capable  of 
distinguishing  among  a  broad  array  of  character  types,  and  recognizing 


mistakes  and  spurious  marks  with  a  high  degree  of  accuracy. 


Gallup  Recommendations: 

Gallup  recommends  that  ARI 
investigate  the  use  of  more 
sophisticated  scanning  equipment. 
Currently,  ARI’s  mail  surveys  require 
respondents  to  indicate  their  answers  by 
filling  in  the  appropriate  bubbles. 


Though  costly,  more  sophisticated 


scanning  equipment  would  permit  greater  flexibility  in  the  method  of 
responding  and  efficiency  in  processing  the  results.  For  example, 
respondents  can  enter  numerical  answers  that  can  be  scanned.  In 


addition,  any  steps  that  permit  the  automation  of  data  processing  can 
significantly  cut  the  time  to  provide  timely  results  that  help  settle 
emerging  policy  issues. 
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Reporting  the 
Results 

The  final  product  of  most 
surveys  is  a  set  of  statistics 
based  on  the  data 
collected,  often 
summarized  as  a  report. 
The  report  should  also 
document  the  methods 
used  to  carry  out  the 
research. 


What  ARI  Does: 

Most  ARI  surveys  report  descriptive  statistics  such  as  percentages, 
means,  and  standard  deviations,  but  each  survey  has  different  reporting 
requirements.  For  example,  while  the  Sample  Survey  of  Military 
Personnel  (SSMP)  usually  reports  simple  percentages,  the  Survey  of  New 
Recruit  Values  (SNRV)  also  includes  a  factor  analysis  of  the  results. 
There  are  very  few  reporting  elements  consistent  across  ARI  report 
types. 


What  Other  Organizations  Do: 

Most  of  the  surveys  Gallup  examined  produce  a  set  of  statistics  derived 
from  the  data  at  hand,  often  summarized  in  a  final  report.  Generally,  the 
reporting  process  also  documents  the  methods  used  to  conduct  the 
survey.  Some  organizations,  including  Gallup,  deliver  reports  in  a  highly 
standardized  format  with  relatively  little  customized  analysis  while 
others  carry  out  detailed  and  sophisticated  analyses  of  the  data. 


Gallup  Recommendations: 

Because  it  tailors  its  products  to  its  customers,  ARI  has  not  adopted  a 
uniform  format  for  documenting  its  surveys.  Gallup  recommends  that  a 
minimal  methodology  report  be  routinely  produced  for  each  ARI  study, 
covering  such  basic  issues  as  sample  design,  questionnaire  design,  the 
method  of  data  collection,  the  response  rate,  and  any  special  procedures 
used  in  developing  survey  estimates  (such  as  weights  or  imputation  of 
missing  values). 


Morale 


Sui*vey  says  it  has  hit  rock  bottom,  citing  more 
missions,  fewer  soldiers,  lower  retirement  pay 


Tt.103  JWAJT 

Ilow  low  can  mornk  go? 

Not  too  much  below  ISWs  reL>. 
'livoiy  Unmiitable  level,  or  so  the 
.^tny  hopes. 

,  Soldiers’  inorule  —  loosely  de- 
Gned  n,'  job  satisfijction  and  quali- 
;ly  of  life  —  hit  took  bottom  in  late 
■1993  compnred  to  oilier  |K>st-Gulf 


'We’re  asking  soldiers  and  their  fairaiies 
to  do  an  awful  lot— more  than  they 
used  to.  This  is  a  new  kind 
of  era  we’re  in/ 

Patrick  T.HKsKsr 

ASST.  AKJW  SCCftCTAKV.  .MaSIOVI-ER  AND  An’AlKS 


Survey  says . 

Based  on  your  Army  experience,  satisfird  aie  you  wilt)  Ids  loilowing 
(Tlioss  respondtrig  “satisfied-  or  “very  safisfied”  in  tall  surveys) 


1997 


-it  .... 

1993 
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New  Directions 


Summary  of  Gallup  Recommendations 


Gallup ’s  report  included 
suggestions  for  improving 
ARI’s  survey  response 
rates,  standardizing  its 
pretesting  procedures, 
adopting  new  data 
collection  technologies, 
creating  a  quick 
turnaround  capability, 
standardizing  deliverables, 
and  obtaining  systematic 
customer  feedback. 


Gallup’s  review  indicates  that  ARI  is  doing  a  good  job.  ARI  carries  out 
surveys  at  a  relatively  low  cost,  uses  methodologically  sound  procedures, 
and  achieves  reasonably  high  response  rates.  ARI  has  moved  with  the 
times  and  is  increasingly  relying  on  computer-assisted  data  collection. 
Their  customers  at  a  wide  range  of  agencies  find  the  data  useful  and  are 
highly  satisfied  with  ARI’s  work.  ARI’s  surveys  exhibit  the  same  sorts 
of  problems  (such  as  response  rate  and  coverage  problems)  as  the 
benchmark  surveys  Gallup  examined. 


Gallup  presented  recommendations  in  six  areas.  ARI  should: 

•  take  steps  to  improve  its  response  rates; 

•  strengthen  and  standardize  its  pretesting  procedures; 

•  continue  to  adopt  new  survey  technologies; 

•  explore  the  creation  of  a  quick-turnaround  capability  to  respond  to 
emerging  policy  issues; 

•  adopt  minimal  standards  for  the  documentation  provided  to  its 
customers; 

•  solicit  systematic  customer  feedback. 


“Our  review  indicates  that  ARI  is  doing  a  good  job.  It 
carries  out  surveys  at  a  relatively  low  cost,  uses 
methodologically  sound  procedures,  and  achieves  reasonably 
high  response  rates.” 

Gallup  report  to  ARI 
May  3,  1999 
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